An Intelligent Marshaling Based on Transfer Distance of Containers Using a New Reinforcement Learning for Logistics

Recent shipping amount in maritime transportation keeps growing, and efficient material handling operations at marine ports becomes important issue. In many cases, containers are used for transportation of cargos, and thus the growth of shipping amount leads to the growth of the number of containers. In a marine port, containers are shifted between seaborn and landside transportation at container yard terminal. Espesially, shifting containers from landside into a vessel is highly complex, includingmany constraints and sensitive parameters. In addition, the complexity grows at an exponential rate according to the linear growth of the number of containers. Thus, the material hadling operation occupy a large part of the total run time of shipping at container terminals. This chapter addresses to improve throughput of the material handling operations for loading container into a vessel by using reinforcement learning. Commonly, each container in a vessel has its own position determined by the destination, weight, owner, and so on (Günther & Kim, 2005). Thus, the containers have to be loaded into a vessel in a certain desired order because they cannot be rearranged in the ship. Therefore, containers must be rearranged before loading if the initial layout is different from the desired layout. Containers carried into the terminal are stacked randomly in a certain area called bay and a set of bays are called yard. The rearrangement process conducted within a bay is called marshaling. In the problem, the number of stacks in each bay is predetermined and the maximum number of containers in a stack is limited. Containers are moved by a transfer crane and the destination stack for the container in a bay is selected from the stacks being in the same bay. In this case, a long series of container movements is often required to achieve a desired layout, and results that are derived from similar initial layouts can be quite different. Problems of this type have been solved by using techniques of optimization, such as genetic algorithm (GA) and multi agent method (Koza, 1992; Minagawa & Kakazu, 1997). These methods can successfuly yield some solutions for block stacking problems. However, they adopt the environmental model different from the marshaling process, and cannot be applied directly to generate marshaling plan to obtain the desired layout of containers. Another candidate for solving the problem is the reinforcement learning (Watkins & Dayan, 1992), which is known to be effective for learning under unknown environment that has the Markov Property. The Q-learning, one of the realization algorithm for the reinforcement learning, can be applied to generate marshaling plan, with evaluation-values for pairs of the 22